Experiences with Tracing Causality in Networked Services

Unlike device-centric monitoring, task-centric tracing enables an operator to causally trace the complete execution of a networked system across the boundaries of applications, protocols, and administrative domains. In this paper, we argue that causal, end-to-end tracing should be an integral part of network services. Moreover, it is not fundamentally difficult to achieve, given a primitive that propagates task metadata alongside logical execution and communication paths. X-Trace is a framework that relies on such propagation to provide comprehensive causal tracing. We report on our experience integrating X-Trace into several production networked services—including 802.1X authentication, Web content distribution, and DNS-based replica selection—to illustrate benefits of causal tracing, and to discuss the instrumentation of different protocols and component architectures. We highlight the challenges we encountered and techniques we developed to better integrate causal tracing into network services.